The present disclosure relates to methods and systems for creating reference data, such as a reference image, for use with a computer test description for testing of computer programs, such as software applications.
In recent times, more and more computer programs are being developed and used to facilitate aspects of modern life. As computing power and demand for new software features increase, so does the complexity of software products and the code that drives them. This includes increased inter-operability with different devices and software applications. As the computer application and device market becomes more competitive, the number of different device types and operating systems also increases. A program developer is therefore often faced with various devices using various communication protocols and various other software applications to take into account when developing new software. As a result, the intended software product becomes increasingly prone to errors, flaws, failures, or faults (otherwise known as “bugs”) in the program. Typically, these bugs are only discovered when the software is run. The testing of a software product, pre-launch, is therefore important to a software developer.
Software test engineers commonly testing the functionality and behavior of a program both pre and post launch. When performing testing, it is desirable to test out the software on a large number of devices and operating systems to ensure the product is ready for widespread sale for (and can be supported by) all types of computer system. A program which works well on one type of device may experience difficulties running on another type of device. The testing can therefore be a time consuming exercise. Accordingly, the test engineer may want to test multiple devices without having to physically interface with each System Under Test (SUT). Typically the same test will need to be executed a number of different times on the same operating system. Furthermore, when having to perform the same test on a number of different devices, the test engineer would ideally want to run the same sequence of steps without actually having to input each and every step for every test.
One of the most important times to test software is when new versions of the software are released. When such new versions are released, a development team typically creates a “candidate” version of the software. The software testers then test it to find bugs and send the candidate version back to the development team for improvement. The development team then creates a new “candidate”. The testers then re-test it to find further bugs and again send it back to development. This loop continues until the software works correctly and it is delivered to customers.
At some further point in time, the development team will typically add some new features to the existing program. The testers then not only have to test the new features, but also that the old features have not broken (i.e. ceased to operate as desired) with the introduction of and/or interaction with the changes. This is called “regression testing”. Therefore over the life of a software product, a single test case will be executed 10s, 100s, possibly 1000s of times.
Test programs are known to test the functionality of an SUT. Typically the test program is run upon a computing device or computing system that is remote from the SUT. The computer system running the test program is however in communication with the SUT so that the test program can interact with the SUT. The SUT may be, for example, a PC running a version of Microsoft Word™ upon a Windows 7™ operating system.
Test programs typically test an SUT by interacting with the SUT and validating the state of the SUT according to an input test description. Test descriptions may also be referred to as automated test descriptions. Test descriptions can take many forms including but not limited to text instructions, software in source code or binary form, work-flows in a diagrammatical form (such as a flow chart). Typically, test descriptions contain lines of text or other characters that when executed by the test program cause the SUT to perform particular actions or execute functions. Therefore executing (or “running”) the test description through the test program allows the system under test to be tested automatically, without requiring a test engineer to provide input to the testing process, other than initiating it.
Some test programs that use automated test descriptions are known. One test program that helps software engineers automate the testing process is EggPlant™ by TestPlant™. The EggPlant™ software tool allows a user to create and subsequently execute an automated test description to test a large number of SUTs. This test program interacts and controls an SUT by receiving and analyzing images of the SUT Graphical User Interface (GUI).
Test programs such as EggPlant™ that utilize GUI images operate by identifying expected image objects in a GUI image that relate to executable functions that the test description may wish to identify and interact with or indicate the status of the SUT. Once the expected image object has been identified in the GUI image, the test program description can move to the next step in the test. The identification of the expected image object is performed using image analysis. The image analysis searches the GUI image for the expected image object using an existing reference image that is similar or identical to an expected image object. When a match is found, the expected image object is identified. Examples of image objects may include: an image of a button on the GUI that when “clicked” would open a dialogue box, or the presence of a particular graphical object that signified the completion of a particular action performed by the target program. The test program may use image analysis techniques to perform these comparisons.
Reference images are typically created when generating the test description. When generating the test description the user typically uses the test program in a “description creation mode,” to navigate an SUT. Typically the navigation involves manually testing a particular SUT. As the user performs the testing steps, the information about the steps is entered into the test description. Thus when the test description is completed and subsequently run automatically by the test program to test a further SUT, the same testing steps are performed upon the further SUT.
The test description is typically created either manually and/or automatically by a test engineer using the test program. Where commands in the test description require the use of reference images as described above, the user creating the test description manually selects the desired portion of a GUI image (taken from the SUT being tested) using an image selection tool whereby the selected portion of the GUI image is saved as the reference image. Manually selecting images can become time consuming for the test engineer creating the test description.
Issues may also arise when new versions of the target software are made that cause a change in the graphical representation of the expected image object because the existing reference image may appear significantly different. Therefore the image analysis to identify the expected image object using an existing reference image may fail.